Modular systems and methods for selectively enabling cloud-based assistive technologies

ABSTRACT

Systems and methods are disclosed for manually and programmatically remediating websites to thereby facilitate website navigation by people with diverse abilities. For example, an administrator portal is provided for simplified, form-based creation and deployment of remediation code, and a machine learning system is utilized to create and suggest remediations based on past remediation history. Voice command systems and portable document format (PDF) remediation techniques are also provided for improving the accessibility of such websites.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/991,671, filed Aug. 12, 2020, and titled “MODULAR SYSTEMS AND METHODSFOR SELECTIVELY ENABLING CLOUD-BASED ASSISTIVE TECHNOLOGIES,” which is acontinuation-in-part of U.S. patent application Ser. No. 16/430,210,filed Jun. 3, 2019, and titled “MODULAR SYSTEMS AND METHODS FORSELECTIVELY ENABLING CLOUD-BASED ASSISTIVE TECHNOLOGIES,” which is acontinuation of U.S. Patent Application No. 15/074,818, filed Mar. 18,2016, now U.S. Pat. No. 10,444,934, and titled “MODULAR SYSTEMS ANDMETHODS FOR SELECTIVELY ENABLING CLOUD-BASED ASSISTIVE TECHNOLOGIES.”U.S. patent application Ser. No. 16/991,671, filed Aug. 12, 2020, andtitled “MODULAR SYSTEMS AND METHODS FOR SELECTIVELY ENABLING CLOUD-BASEDASSISTIVE TECHNOLOGIES” is also a continuation-in-part of U.S. patentapplication Ser. No. 16/533,568, filed Aug. 6, 2019, now U.S. Pat. No.10,762,280, and titled “SYSTEMS, DEVICES, AND METHODS FOR FACILITATINGWEBSITE REMEDIATION AND PROMOTING ASSISTIVE TECHNOLOGIES,” which is acontinuation of U.S. patent application Ser. No. 15/999,116, filed Aug.16, 2018, now U.S. Pat. No. 10,423,709, and titled “SYSTEMS, DEVICES,AND METHODS FOR AUTOMATED AND PROGRAMMATIC CREATION AND DEPLOYMENT OFREMEDIATIONS TO NON-COMPLIANT WEB PAGES OR USER INTERFACES.” Each of theforegoing applications is hereby incorporated by reference in itsentirety.

TECHNICAL FIELD

The embodiments herein relate generally to systems, devices, and methodsfor selectively enabling cloud-based assistive technologies and, moreparticularly, to techniques for remediating websites and other graphicaluser interfaces to enhance the user experience for users having diverseabilities.

BACKGROUND

Various forms of assistive technologies, such as screen readerapplications, voice command devices, and the like, have proven useful tovisually impaired, color impaired, low vision, dyslexic, illiterate, andlearning-disabled individuals, particularly with respect to computeraccessibility.

Screen readers, for example, are used to translate textual and graphicalinformation displayed on a screen and re-present it to the user usingsynthesized speech, sound icons, and/or a refreshable Braille outputdevice. Screen readers may be incorporated into the operating system asnative applications or may be packaged as separate commercial products.

Presently known screen readers, voice command systems, and otherassistive technologies are disadvantageous, however, because of theirlimited ability to remediate sites which do not comply with industryrecognized best practices, such as the Web Content AccessibilityGuidelines (WCAG) 2.0, Accessible Rich Internet Applications (WAI-ARIA),Authoring Tool Accessibility Guidelines (ATAG), Section 508 Standards &Technical Requirements, and other existing and forthcoming national andinternational standards and guidelines.

New systems, devices, and methods are thus needed which overcome thelimitations of assistive technologies.

SUMMARY

Various embodiments herein relate to systems, devices, and methods forproviding an administrator portal for simplified, form-based creationand deployment of remediation code. Some embodiments herein relate tosystems, devices, and methods for a machine learning platform trained tocreate and propose remediations and/or associated code based on, forexample, past remediation history. Some embodiments herein relate tosystems, devices, and methods for determining a set of remediationsapplicable to multiple web pages by, for example, identifying a templateapplicable to those web pages (e.g., based on DOM structure, URLstructure, and/or accessibility test results). Some embodiments hereinrelate to systems, devices, and methods for monitoring the frequencywith which web pages on a website are accessed (e.g., via an embeddedtracking code configured to cause an analytics event) and prioritizingthe remediation of the web pages accordingly. Some embodiments hereinrelate to systems, devices, and methods for adjusting the frequency withwhich web pages are scanned for remediation issues based on, forexample, an estimated rate at which the web pages undergo a change ofcontent. Some embodiments herein relate to systems, devices, and methodsfor personal information storage that allows a user to store and recalldata (e.g., form fill-out data, personal notes, and/or custom“overrides” and shortcuts) using voice-based interactions. Someembodiments herein relate to providing a voice command platform thatallows a user to perform a series of interactions (e.g., across multipleweb pages) by issuing a single voice command. Some embodiments hereinrelate to systems, devices, and methods of a voice command platform thatallows a user to view a guided tutorial that includes a series ofinteractions by issuing a single voice command. Some embodiments hereinrelate to systems, devices, and methods for dynamically routing voicecommands to one or more natural language processing (NLP) and automaticspeech recognition (ASR) systems based on context. Some embodimentsherein relate to systems, devices, and methods for providing numericallabels adjacent to interactive elements on a web page to aid inselecting those interactive elements via a voice command. Someembodiments herein relate to systems, devices, and methods for providingfor forwarding voice command requests to a selected service providerbased on pre-populated user preference data. Some embodiments hereinrelate to systems, devices, and methods for enhanced accessibility ofPDF documents.

In some embodiments, described herein are systems, devices, or methodsfor programmatic creation and deployment of remediations tonon-compliant web pages or user interfaces, the system comprising: oneor more remediation code databases configured to store a plurality ofremediation code blocks; an integrated JavaScript code base for creatingclient-side interactivity with one or more web sites or user interfacescomprising a hypertext markup language (HTML) document object model(DOM), wherein the DOM comprises one or more nodes, the one or morenodes organized into a DOM tree structure; one or more computer readablestorage devices configured to store a plurality of computer executableinstructions; and one or more hardware computer processors incommunication with the one or more computer readable storage devices andconfigured to execute the plurality of computer executable instructionsin order to cause the system to: receive, from a user device, a requestto access a web page associated with the one or more web sites or userinterfaces; dynamically crawl, using a configuration file, the one ormore web sites or user interfaces to generate a list of uniform resourcelocators (URLs) to be included as part of a compliance assessment of theone or more web sites or user interfaces; evaluate each of the one ormore web pages associated with the generated list of URLs to identifyone or more compliance issues, wherein the evaluation comprises loadingeach of the one or more web pages in a headless browser, applying theintegrated Javascript code base, and executing a series of compliancetests; access, from the one or more remediation code databases, one ormore remediation code blocks, each remediation code block correspondingto an identified compliance issue; programmatically apply the one ormore remediation code blocks to the corresponding identified one or morecompliance issues; and transmit, to the user device, for display, on theuser device, the requested web page to which the one or more remediationcode blocks have been applied; wherein application of the one or moreremediation code blocks manipulates the DOM, causing a correction orenhancement of the one or more web sites or user interfaces upon loadingand rendering.

In some embodiments, described herein are systems, devices, or methods,wherein the crawl of the one or more web sites or user interfaces isbased on one or more scanning parameters, wherein the scanningparameters are manually or automatically configurable, and wherein thescanning parameters comprise at least one of maximum crawl time, levelof recursion, percentage of pages catalogued, configurable threshold,and page depth.

In some embodiments, described herein are systems, devices, or methods,wherein the configuration file comprises one or more configurationoptions, the configuration options comprising configuration of at leastone of: a user agent, browser dimensions, starting page, inclusion ofsubdomains, inclusion of cookies, and usage of a defined site map.

In some embodiments, described herein are systems, devices, or methods,wherein the system is further caused to perform a link extractionprocess, wherein the link extraction process detects one or more URLswithin the DOM to be added to the generated list of URLs.

In some embodiments, described herein are systems, devices, or methods,wherein the compliance tests assess one or more testable successcriteria as supported by Web Content Accessibility Guidelines (WCAG).

In some embodiments, described herein are systems, devices, or methods,wherein the compliance tests are automatically or manually altered basedon evolving web accessibility standards.

In some embodiments, described herein are systems, devices, or methods,wherein the compliance tests determine whether or not one or morespecific use-cases are accommodated within the hypertext markup language(HTML) document object model (DOM).

In some embodiments, described herein are systems, devices, or methods,wherein the one or more hardware computer processors in communicationwith the one or more computer readable storage devices is furtherconfigured to execute the plurality of computer executable instructionsin order to cause the system to: determine a confidence rating for eachof the one or more identified compliance issues.

In some embodiments, described herein are systems, devices, or methods,wherein the one or more hardware computer processors in communicationwith the one or more computer readable storage devices is furtherconfigured to execute the plurality of computer executable instructionsin order to cause the system to: determine a confidence rating and/orseverity rating for each of the programmatic applications of the one ormore remediation code blocks to the corresponding one or more identifiedcompliance issues.

In some embodiments, described herein are systems, devices, or methods,wherein the programmatic application of the one or more remediation codeblocks is combined with manual remediations to completely remove the oneor more identified compliance issues.

In some embodiments, described herein are systems, devices, or methods,wherein the one or more identified compliance issues comprise at leastone of the following: language attribute not set, INPUT missing label,empty anchor, heading missing text, text alternatives missing fromnon-text content, unnecessary duplication of link description, skip-tolink not present, and links do not warn user before opening a newwindow.

In some embodiments, described herein are systems, devices, or methods,further comprising a real-time auto-detection and audio enablement(RADAE) engine configured to detect web page content and element types.

In some embodiments, described herein are systems, devices, or methods,further comprising a machine-learning engine configured to learn fromthe programmatic application of the one or more remediation code blocksto the corresponding one or more identified compliance issues, allowingit to create and suggest future programmatic applications.

In some embodiments, described herein are systems, devices, or methods,wherein the machine learning engine learns using supervised,unsupervised, semi-supervised, or reinforcement learning paradigms.

In some embodiments, described herein are systems, devices, or methods,wherein the one or more remediation code databases comprise dynamic linklibraries (DLL) or JavaScript object notation (JSON) format databases.

In some embodiments, described herein are systems, devices, or methodsfor rendering one or more web pages or user interfaces enhanced by anintegrated JavaScript, the system comprising: an accessibility serverconfigured to host one or more accessibility processes, wherein theaccessibility processes comprise at least one of real-timeauto-discovery, audio enablement, or text to speech processing; one ormore computer readable storage devices configured to store a pluralityof computer executable instructions; and one or more hardware computerprocessors in communication with the one or more computer readablestorage devices and configured to execute the plurality of computerexecutable instructions in order to cause the system to: receive arequest to access a web page from a web browser; retrieve the web page;cause to execute, within the web browser, an integrated JavaScript,wherein executing the integrated JavaScript comprises retrieving,through a network connection to the accessibility server, at least oneof the one or more accessibility processes for execution by the webbrowser; receive a request from the web browser and retrieve remediationcode from the JavaScript, the remediation code configured to solve oneor more compliance issues of the web page; cause the web browser torender the web page; and cause to execute, within the web browser, theremediation code, wherein execution of the remediation code transforms adocument object model (DOM) of the web page into an accessible state.

In some embodiments, described herein are systems, devices, or methodsfor form-based creation and deployment of remediations to non-compliantweb pages or user interfaces, the system comprising: one or moreremediation code databases configured to store a plurality ofremediation code blocks; an integrated JavaScript code base for creatingclient-side interactivity with one or more web sites or user interfacescomprising a hypertext markup language (HTML) document object model(DOM), wherein the DOM comprises one or more nodes, the one or morenodes comprising visual elements tagged with alternative textdescriptions organized into a DOM tree structure; one or more computerreadable storage devices configured to store a plurality of computerexecutable instructions; and one or more hardware computer processors incommunication with the one or more computer readable storage devices andconfigured to execute the plurality of computer executable instructionsin order to cause the system to: provide a dynamic user interfaceconfigured to allow creation and application one or more acceleratedremediations, wherein the one or more accelerated remediations compriseremediation code and metadata; cause display, through the dynamic userinterface, the visual elements of the one or more web sites or userinterfaces; evaluate each of the one or more web sites to identify oneor more compliance issues, wherein the evaluation comprises loading eachof the one or more sites in a headless browser, applying the integratedJavaScript code base, and executing a series of compliance tests;access, from the one or more remediation code databases, one or moreremediation code blocks, each remediation code block corresponding to anidentified compliance issue; populate a web-based form configured tocollect the metadata, wherein the metadata comprises remediation contextinformation; programmatically generate and store one or more acceleratedremediation code blocks within the one or more remediation codedatabases; receive, from the integrated JavaScript code base, a requestfor one or more accelerated remediation code blocks corresponding to oneor more web sites or user interfaces accessed by a user web browser;deliver, to the user web browser, the one or more acceleratedremediation code blocks corresponding to one or more web sites or userinterfaces accessed by a user; and execute the one or more acceleratedremediation code blocks, wherein execution comprises modification of theDOM; wherein all of the accelerated remediation code blockscorresponding to one of the one or more web sites are stored in the sameremediation code database.

In some embodiments, described herein are systems, devices, or methods,further comprising a machine learning engine, the machine learningengine configured to: compare the identified compliance issues with oneor more previously identified and resolved compliance issues; andascertain one or more previously executed remediation code blocks thatmay be applicable to the identified compliance issue from the one ormore remediation code databases.

In some embodiments, described herein are systems, devices, or methods,wherein the one or more accelerated remediations can be applied to oneor more of the following compliance issues: inadequate ALT-text, linkimages with missing or inadequate ALT-text, input images with missing orinadequate ALT-text, iFrames missing titles, disordered headers, headinglevel issues, inputs missing labels, or links missing labels.

In some embodiments, described herein are systems, devices, or methodsfor identifying structural patterns across a plurality of web pages anddetermining a set of remediations that can be applied to the pluralityof web pages, the method comprising: identifying, by a computer system,within a first document object model (DOM) corresponding to a first webpage of the plurality of web pages, a first hierarchal structure,wherein the first web page comprises a plurality of elements, whereineach hierarchy of the first hierarchal structure comprises one or moreelements having commonalities in placement, class, or appearance;evaluating, by the computer system, the first web page to identify oneor more compliance issues, wherein the evaluation comprises loading thefirst web page in a headless browser, applying an integrated JavaScriptcode base, and executing a series of compliance tests; determining, bythe computer system, one or more remediations to resolve the one or moreidentified compliance issues of the first web page; storing, by thecomputer system, the one or more remediations in a remediation database;identifying, by the computer system, within a second document objectmodel (DOM) corresponding to a second web page of the plurality of webpages, a second hierarchal structure, wherein the second hierarchalstructure comprises a substantially identical structural pattern to thefirst hierarchal structure; and applying, by the computer system, theone or more remediations to the second web page, wherein the computersystem comprises a computer processor and an electronic storage medium.

In some embodiments, described herein are systems, devices, or methodsfor website accessibility monitoring, the system comprising: one or moreanalytics database engines configured to store one or more page loadevents received from a plurality of websites, wherein each of theplurality of websites comprises tracking code that activates a page loadevent upon a user loading the website, and wherein the page load eventcomprises a uniform resource locator (URL) and the date and time of theloading of the website; a data accessibility scanning and testing engineconfigured to scan one or more of the plurality of websites and performone or more accessibility tests; one or more testing database enginesconfigured to store testing data, wherein the testing data is related tothe one or more accessibility tests; one or more computer readablestorage devices configured to store a plurality of computer executableinstructions; and one or more hardware computer processors incommunication with the one or more computer readable storage devices andconfigured to execute the plurality of computer executable instructionsin order to cause the system to: request, from the one or more analyticsdatabase engine, one or more URLs of the one or more stored page loadevents; load, by the data accessibility scanning and testing engineusing a headless browser, the one or more URLs; perform the one or moreaccessibility tests on each of the one or more URLs to generate testingdata; store, in the one or more testing database engines, the testingdata corresponding to each of the one or more URLs; extract, from theone or more testing database engines, metadata corresponding to thetesting data corresponding to each of the one or more URLs; and generatea report of the testing data and/or metadata, wherein the reportprioritizes one or more results of the one or more accessibility tests.

In some embodiments, described herein are systems, devices, or methods,wherein requesting the one or more URLs comprises one or more URLfiltering criteria.

In some embodiments, described herein are systems, devices, or methods,wherein the testing data corresponding to each of the one or more URLscomprises one or more of: the URL, a hash of the URL, the accessibilitytests passed by each element of a web page located by the URL, the testsfailed by each element, or an aggregate count of accessibility testspasses and failures.

In some embodiments, described herein are systems, devices, or methods,wherein the one or more results of the one or more accessibility testsare prioritized based at least in part on the severity of the one ormore results on an accessibility metric, a number of web pages on whichthe one or more results occurs, and a number of times each web page hasbeen accessed.

In some embodiments, described herein are systems, devices, or methods,wherein the data accessibility scanning and testing engine is furtherconfigured to automatically determine a frequency to scan a website ofthe plurality of websites.

In some embodiments, described herein are systems, devices, or methods,wherein determining the frequency to scan the website comprises trackingand/or estimating the frequency the website is altered.

In some embodiments, described herein are systems, devices, or methods,wherein tracking and/or estimating the frequency the website is alteredcomprises one or more or of: monitoring a combination of data points,comparing scanned page HTML or a stored hash to historical versions ofthe same, comparing changes in failure conditions, comparing changes infixed errors and risks, fixed errors and risks, or comparing changes inaccess patterns.

In some embodiments, described herein are systems, devices, or methods,wherein the combination of data points comprises HTTP header metadata.

In some embodiments, described herein are systems, devices, or methods,wherein the failure conditions comprise a total number of identifiederrors and risks.

In some embodiments, described herein are systems, devices, or methodsfor filling a web form utilizing a personal information storage systemvia voice-based interaction, the method comprising: creating, by acomputer system based at least in part on a user input via a user accesspoint, a user profile, wherein the user profile is linked to a useridentifier; storing, by the computer system, the user profile in one ormore user profile databases; receiving, by the computer system via theuser access point through a voice platform, a voice command from theuser, instructing the personal information storage system to store userinformation; storing, by the computer system, the user information inthe user profile; triggering, by the computer system using web browsertechnologies and scripting techniques, a voice interaction toauthenticate a user, permitting access into a secure, password protectedenvironment; receiving, by the computer system from the user, a voicecommand from the user, instructing the personal information storagesystem to insert the user identifier or user information into a fillableweb form; and causing, by the computer system, insertion of the useridentifier or user information into the fillable web form, wherein thecomputer system comprises a computer processor and an electronic storagemedium.

In some embodiments, described herein are systems, devices, or methods,wherein the user identifier comprises one or more of: an e-mail addressor one or more third-party single sign-ons (SSOs).

In some embodiments, described herein are systems, devices, or methods,wherein the user information comprises one or more of: home address,work address, phone numbers, banking information, website addresses,password manager information, payment card numbers and expiration dates,email addresses, salutations, or social security numbers.

In some embodiments, described herein are systems, devices, or methods,wherein the user information comprises free-form information.

In some embodiments, described herein are systems, devices, or methods,wherein the user information comprises a personalization override,wherein the personalization override comprises a first alphanumericalstring that corresponds to a voice-command comprising a second, uniquealphanumerical string.

In some embodiments, described herein are systems, devices, or methodsfor executing a series of computer functions in response to a singleuser voice-command, the method comprising: receiving, by a voice commandplatform from a user via a user access point, a voice command;converting, by a computer system, the voice command to a user intent;retrieving, by the computer system from one or more intent databases, aseries of computer functions corresponding to the user intent; andcausing, by the computer system, execution of the series of computerfunctions, wherein the computer system comprises a computer processorand an electronic storage medium.

In some embodiments, described herein are systems, devices, or methods,wherein converting the voice command to the user intent comprisestransmitting the voice command to one or more automatic speechrecognition (ASR) and/or natural language processing (NLP) systems.

In some embodiments, described herein are systems, devices, or methods,wherein causing execution of the series of computer functions comprisesprogrammatically emulating the actions that the user would take tomanually complete the series of computer functions.

In some embodiments, described herein are systems, devices, or methods,wherein the series of computer functions are executed across multiplepages or within multiple views within a Single Page Application (SPA).

In some embodiments, described herein are systems, devices, or methods,wherein causing execution of the series of computer functions comprisesproviding the user with a visual indicator or other assistive technologyto indicate a series of user actions which sequentially complete theseries of computer functions.

In some embodiments, described herein are systems, devices, or methodsfor dynamically routing voice commands to natural language processing(NLP) and/or automatic speech recognition (ASR) systems, the systemcomprising: a voice engine communicatively coupled to a plurality of NLPand/or ASR systems and configured to route voice commands to one or moreof the plurality of NLP and/or ASR systems; one or more computerreadable storage devices configured to store a plurality of computerexecutable instructions; and one or more hardware computer processors incommunication with the one or more computer readable storage devices andconfigured to execute the plurality of computer executable instructionsin order to cause the system to: receive, from a user via a user accesspoint, a voice command; determine, by the voice engine based oncontextual data, a first NLP and/or ASR system of the plurality of NLPand/or ASR systems to which the voice command is to be sent forprocessing; route the voice command to the first NLP and/or ASR system;receive, from the first NLP and/or ASR system, a first resulting outputintent; and cause execution of a computer function based on the receivedfirst resulting output intent.

In some embodiments, described herein are systems, devices, or methods,wherein the system is further caused to: determine, by the voice enginebased at least in part on the contextual data and the first resultingoutput intent, a second NLP and/or ASR system of the plurality of NLPand/or ASR systems to which the voice command and/or first resultingoutput intent is to be sent for processing; receive, from the second NLPand/or ASR system, a second resulting output intent; compare, by thevoice engine, the first resulting output intent and the second resultingoutput intent; dynamically determine a final intent to be executed basedon the received first resulting output intent, the second resultingoutput intent, and one or more internal intent rules; and causeexecution of a computer function based on the determined final intent.

In some embodiments, described herein are systems, devices, or methods,wherein the system is further caused to: prompt the user to providecontextual data or additional information based on the first resultingintent received from the first NLP and/or ASR system; determine, by thevoice engine based on the user-provided contextual data and the firstresulting output intent, a second NLP and/or ASR system of the pluralityof NLP and/or ASR systems to which the voice command and/or firstresulting output intent is to be sent for processing; receive, from thesecond NLP and/or ASR system, a second resulting output intent; compare,by the voice engine, the first resulting output intent and the secondresulting output intent; dynamically determine a final intent to beexecuted based on the received first resulting output intent, the secondresulting output intent, and one or more internal intent rules; andcause execution of a computer function based on the determined finalintent.

In some embodiments, described herein are systems, devices, or methods,wherein the contextual data includes one or more of: content of thevoice command, content of a website being viewed by the user, contentsof a preconfigured user profile, date and time of the voice command,length of the voice command, language of the voice command, level ofbackground noise within a voice command audio file, or amplitude of thevoice command.

In some embodiments, described herein are systems, devices, or methodsfor a voice-enabled kiosk, browser, or point-of-sale user interfacesystem, the system comprising: a voice input device configured toreceive user voice commands; an integrated JavaScript engine configuredto facilitate natural language processing (NLP) and/or automatic speechrecognition (ASR) and voice command execution; one or more computerreadable storage devices configured to store a plurality of computerexecutable instructions; and one or more hardware computer processors incommunication with the one or more computer readable storage devices andconfigured to execute the plurality of computer executable instructionsin order to cause the system to: receive, from a user via a user accesspoint, a voice command; route the voice command to a natural languageprocessing (NLP) and/or automatic speech recognition (ASR) forprocessing; receive, from the natural language processing (NLP) and/orautomatic speech recognition (ASR), a text output of the voice command;interpret, by the integrated Javascript engine, a user intent based onthe text output of the voice command received from the natural languageprocessing (NLP) and/or automatic speech recognition (ASR); and causeexecution of a computer function based on the interpreted user intent.

In some embodiments, described herein are systems, devices, or methods,wherein the voice input device is an integrated microphone.

In some embodiments, described herein are systems, devices, or methods,wherein the voice input device is a secondary device, wherein thesecondary deice is paired remotely to one or more of the voice-enabledkiosk, browser, or point-of-sale user interface system.

In some embodiments, described herein are systems, devices, or methods,wherein natural language processing (NLP) and/or automatic speechrecognition (ASR) is at least partially completed internally by theintegrated JavaScript.

In some embodiments, described herein are systems, devices, or methods,wherein natural language processing (NLP) and/or automatic speechrecognition (ASR) is completed by a third-party service.

In some embodiments, described herein are systems, devices, or methods,further comprising a custom function processor configured to interpret avoice command tied to common behaviors, wherein the custom functionprocessor bypasses natural language processing (NLP) and/or automaticspeech recognition (ASR) to generate a user intent.

It should be noted that the various embodiments described herein, whileillustrated in the context of websites, web browsers, and stand-alonekiosks, are not so limited. Those skilled in the art will appreciatethat the systems, devices, and methods described herein may contemplateany paradigm in which assistive technologies may be brought to bear toenhance a user experience.

Moreover, while various embodiments are described as using AEJavaScript, it is understood that any approach which augments existingcode to remediate compliance issues and integrate assistive technologiesto enhance the user experience is contemplated by the embodimentsdescribed herein. In addition, the various software modules describedherein may be implemented in a variety of software languages and/ordatabase structures.

For purposes of this summary, certain aspects, advantages, and novelfeatures of the invention are described herein. It is to be understoodthat not necessarily all such advantages may be achieved in accordancewith any particular embodiment of the invention. Thus, for example,those skilled in the art will recognize that the invention may beembodied or carried out in a manner that achieves one advantage or groupof advantages as taught herein without necessarily achieving otheradvantages as may be taught or suggested herein.

All of these embodiments are intended to be within the scope of theinvention herein disclosed. These and other embodiments will becomereadily apparent to those skilled in the art from the following detaileddescription having reference to the attached figures, the invention notbeing limited to any particular disclosed embodiment(s).

BRIEF DESCRIPTION OF THE DRAWINGS

The features, aspects and advantages of the embodiments of theinventions are described in detail below with reference to the drawingsof various embodiments, which are intended to illustrate and not tolimit the inventions. The drawings comprise the following figures inwhich:

FIG. 1 is a schematic flow diagram of a web crawler process for scanningwebsites in accordance with various embodiments herein;

FIG. 2 is a combined schematic block and schematic flow sequence diagramillustrating exemplary systems, devices, and methods for rendering an AEJavaScript enhanced web page or other user interface components inaccordance with various embodiments herein;

FIG. 3 is a schematic flow diagram illustrating an exemplary remediationdelivery system in accordance with various embodiments herein;

FIG. 4 is a schematic flow diagram illustrating an exemplary “quick fix”remediation system in accordance with various embodiments herein;

FIG. 5 is a schematic flow diagram illustrating an exemplary method ofdelivering a “quick fix” remediation in accordance with variousembodiments herein;

FIG. 6 is a schematic flow diagram illustrating an exemplary remediationprocess incorporating a machine learning component in accordance withvarious embodiments herein;

FIG. 7 is a schematic block diagram illustrating an exemplary templateidentification scheme utilizing a DOM structure in accordance withvarious embodiments herein;

FIG. 8 is a listing of URLs illustrating an exemplary templateidentification protocol utilizing URL structures in accordance withvarious embodiments herein;

FIG. 9 is a schematic flow diagram illustrating an exemplary monitoringsystem in accordance with various embodiments herein;

FIG. 10 illustrates an example output table and impact analysisperformed in connection with a monitoring system in accordance withvarious embodiments herein;

FIG. 11 is a schematic flow diagram illustrating an exemplary automaticscan frequency adjustment system in accordance with various embodimentsherein;

FIG. 12 is a schematic flow diagram illustrating the use of an exemplarypersonal information store in connection with voice-based interactionsin accordance with various embodiments herein;

FIG. 13 is a schematic block diagram illustrating exemplary seriescommands in connection with voice-based interactions in accordance withvarious embodiments herein;

FIG. 14 is a schematic flow diagram illustrating an exemplary system fordynamically routing voice commands to an appropriate natural languageprocessing (NLP) and/or automatic speech recognition (ASR) system inaccordance with various embodiments herein;

FIG. 15 is a flowchart illustrating an exemplary method for dynamicallyrouting voice commands in accordance with various embodiments herein;

FIG. 16 illustrates an exemplary user interface including numericallylabeled elements in accordance with various embodiments herein;

FIG. 17 is a schematic flow diagram illustrating an exemplary system foremploying a middle tier in connection with a voice device in accordancewith various embodiments herein;

FIG. 18 is a schematic flow diagram illustrating an exemplary system foremploying a middle tier in connection with a voice device in accordancewith various embodiments herein;

FIG. 19 is an exemplary web page illustrating the use of an exemplaryportable document format generation tool in accordance with variousembodiments herein.

FIG. 20 is a block diagram depicting an example embodiment of a computersystem configured to run software for implementing one or moreembodiments of the website remediation and assistive technologiessystems, methods, and devices disclosed herein; and

FIG. 21 a flowchart illustrating an exemplary system for a voice-enabledkiosk or point-of-sale user interface in accordance with variousembodiments herein.

DETAILED DESCRIPTION

The following detailed description is merely exemplary in nature and isnot intended to limit the embodiments or the application and uses of theembodiments described herein. Furthermore, there is no intention to bebound by any theory presented in the preceding background or thefollowing detailed description.

Although several embodiments, examples, and illustrations are disclosedbelow, it will be understood by those of ordinary skill in the art thatthe inventions described herein extend beyond the specifically disclosedembodiments, examples, and illustrations and includes other uses of theinventions and obvious modifications and equivalents thereof.Embodiments of the inventions are described with reference to theaccompanying figures, wherein like numerals refer to like elementsthroughout. The terminology used in the description presented herein isnot intended to be interpreted in any limited or restrictive mannersimply because it is being used in conjunction with a detaileddescription of some specific embodiments of the inventions. In addition,embodiments of the inventions can comprise several novel features and nosingle feature is solely responsible for its desirable attributes or isessential to practicing the inventions herein described.

Various embodiments described herein relate to systems, devices, andmethods for bringing websites and other user interfaces into compliancewith applicable standards to thereby facilitate enhanced accessibility.Various embodiments herein allow, for example, developers andnon-technical content publishers to voice-enable their website and webapplications. In some embodiments, the systems, devices, and methodsdescribed herein relate to, for example, a cloud-based dynamic userinterface that provides a practical means of adding voice-controller totheir web-based ecosystems. Some embodiments herein relate to addingvoice-driven controls to any web environment enabled with the JavaScripttechnology described herein. Various embodiments herein provide adynamic user interface to configure custom controls and define customactions to be associated with the anticipated end-user intent suppliedthrough, for example, voice-commands. Presently known screen readers,voice command systems, and other assistive technologies aredisadvantageous because of their limited ability to remediate siteswhich do not comply with industry recognized best practices, such as theWeb Content Accessibility Guidelines (WCAG) 2.0, Accessible RichInternet Applications (WAI-ARIA), Authoring Tool AccessibilityGuidelines (ATAG), Section 508 Standards & Technical Requirements, andother existing and forthcoming national and international standards andguidelines.

In various embodiments, bringing websites and user interfaces intocompliance, such that accessibility functions can be added, isaccomplished by scanning websites and programmatically detecting issuesin a robust and efficient manner, and injecting code into the HTMLdocument object model (DOM) to facilitate both programmatically andmanually remediating noncompliance issues, as described in greaterdetail below. In some embodiments, the system is configured to conductone or more compliance tests. In some embodiments, the tests conductedby the system can be focused on the testable success criteria assupported by the Web Content Accessibility Guidelines (WCAG) 2.0/2.1 orother web accessibility standards. In some embodiments, the testsconducted by the system can be automatically or manually altered basedon evolving web accessibility standards. In some embodiments, the systemmay conduct hundreds of tests to understand, for example, whether or notspecific use-cases (i.e. disability use-cases) are accommodated(accounted for) within the code or the way in which the elements arepresented in the DOM. In some embodiments, not all accessibility elementcriteria may be programmatically testable and in the case of those thatare not, the system may associate remaining testable success criteriawith the pages/elements identified in a crawling or “seeding” process.In some embodiments, the system can be configured to determine aconfidence rating for an identified compliance issue. In someembodiments, an issue may be determined with a confidence rating of100%, such that the system is certain that an issue exists. In someembodiments, compliance issues having a confidence rating above apredetermined threshold may eligible for automatic remediation. In someembodiments, common issues which may be automatically remediated mayinclude, but are not limited to, language attribute not set, INPUTmissing label, empty anchor, heading missing text, text alternativesmissing from non-text content, unnecessary duplication of linkdescription, skip-to link not present, and links do not warn user beforeopening a new window.

More particularly, web pages and other electronic documents, and/ordocuments otherwise accessible through a graphical user interface (GUI)may have an established reading order embodied in the DOM, includingvisual elements that are tagged with alternative text descriptions. Thenodes of the document may be organized in a tree structure, called theDOM tree. In some embodiments, when an HTML page is rendered, a browsermay download the HTML into local memory and may automatically parse itbefore, during, or after displaying the page.

The W3C Document Object Model (DOM), a W3C (World Wide Web Consortium)standard for accessing documents, can comprise a platform andlanguage-neutral interface that may allow programs and scripts (such asJavaScript) to dynamically access and update the content, structure, andstyle of an electronic document. In some embodiments, the HTML DOM maybe a standard object model and programming interface for HTML, and maydefine the properties of some or all HTML elements and the methods foraccessing the HTML elements. In other words, the HTML DOM can act as astandard for how to acquire, change, add, delete, or otherwise interactwith HTML elements.

In the context of various embodiments, objects in the DOM tree may beaddressed and manipulated by an integrated JavaScript code baseavailable from AudioEye, Inc. (AE) at www.audioeye.com, sometimesreferred to herein as the AE JavaScript. The AE JavaScript can createclient-side interactivity with objects in the DOM tree. In someembodiments, the Javascript plug-in can identify and solve digitalaccessibility issues automatically or by allowing a user to manuallyremediate online content. In some embodiments, the system can targetelements of the DOM and add additional or missing attributes to allowweb content to effectively communicate with, for example, anaccessibility API. In some embodiments, additional non-visual elementscan be added to the DOM to improve an assistive technology userexperience. In some embodiments, the system may use a test-centricapproach to fixing accessibility issues and applying remediations. Insome embodiments, remediations can be applied based on a real-timespecific test failure condition. In some embodiments, if no test failureconditions are identified by the system, no remediations are applied. Insome embodiments, tests can be re-run periodically or upon user request.In some embodiments, the system fixes accessibility issues by utilizinga library of automatic remediations. In some embodiments, automaticremediations can be equally applicable to all websites and can beutilized to improve the accessibility of any website. In someembodiments, manual remediations can be custom, site-specific functionsdesigned to remediate unique issues on a given page/site/application. Insome embodiments, automatic remediations may be combined with manualremediations to completely remove accessibility issues with on a webpage. In some embodiments, remediations can be applied by the systemwithout access to the source code of a web page or application to beremediated.

In some embodiments, when an end-user accesses a website enabled withthe techniques described herein, a version of the AE JavaScript is oneof the components that loads on that page. When loaded, the JavaScriptmay inject appropriate compliance fixes (also referred to asremediations) into the page. In some embodiments, screen readers (e.g.JAWS, NVDA, Voice Over, etc.) can read and process the page, leveragingvarious compliance remediation techniques applied to the page.Specifically, the AE JavaScript can check for specific non-conformanceissues and, if detected, programmatically fix them.

In various embodiments, the AE JavaScript may enhance the userexperience using a series of secure communication links with cloud-basedor integrated services, in conjunction with various browser techniques.For example, a real-time auto-detection and audio enablement (RADAE)engine can detect page content and element types. In some embodiments,through cascading style sheet (CSS) selector identification andtechniques similar to x-path, the AE JavaScript may recognize individualHTML entities based on their placement within the DOM and their tag orother identifier. The AE JavaScript may apply rules programmaticallythat make modifications to elements contained within the DOM,facilitating enhanced user accessibility. In some embodiments, only pagemarkup is altered, not the underlying content. In this way, the originalversions of websites remain unaltered when viewed without assistivetechnologies. In other embodiments, both the page markup and theunderlying content can be altered.

In accordance with some embodiments, the systems herein may crawl(spider) websites and gather pages to be analyzed in an evaluationphase. In this regard, various scanning parameters including, forexample maximum crawl time and level of recursion (sometimes referred toas page depth) may be configurable by a system administrator orautomatically configured. However, in some embodiments, web crawling isonly one way to establish an inventory of pages/sites/elements that areto be tested, tracked, and audited via the platform. In someembodiments, such seeding may occur from the results of the JavaScriptbeing embedded on a customer or user site in which case the site trafficand the pages which may be “hit” or accessed by end-users of those sitesembedding the AudioEye JavaScript may be stored and used to seed thesystem with the inventory that is intended to be tracked/audited. Insome embodiments, other methods may also be used to “push” informationinto the system to be tested. In some embodiments, information may betested in a browser, and the test results may be introduced into anauthorized account, which can be retrieved by an authorized user (e.g. auser uses a browser plug-in to conduct tests in their browser session ofan authenticate session within a Single Page Application (SPA) and testresults are sent programmatically via, for example, an API, to apre-defined account within the system for futureretrieval/archival/reporting/auditing/etc.).

In some embodiments, the system can individually evaluate each URL(page) for WCAG or similar compliance issues. For each identifiedcompliance issue, the system may programmatically fix the issue by usinga platform-default block of remediation code (also referred to herein asauto-remediation), programmatically fix the issue by using amanually-generated block of remediation code (also referred to herein asmanual remediation), or alert a human operator to do so. Forprogrammatic as well as manual fixes, the system may utilize one or morecode blocks from one or more remediation libraries, for example, in adynamic link library (DLL), JavaScript object notation (JSON) formatdatabase, or other structured storage mechanism. In some embodiments,the system may be configured to generate and display a dynamic userinterface to a user to manage the remediations. The dynamic userinterface may have enhanced usability such that anon-coder/non-developer can simply use a web form to fix issues ofaccessibility. In some embodiments, the dynamic user interface maychange and react according to specific remediations to be performed. Insome embodiments, the dynamic user interface may be able to facilitateall types of remediations. However, in some embodiments, certain issuesof accessibility are conducive to being remediated through the dynamicuser friendly interfaces. In some embodiments, non-conduciveremediations can either be remediated through automation or manualremediation coded by, for example, a Subject Matter Expert/JavaScriptEngineer. In some embodiments, in the case of Auto-Remediation, thesystem may complete remediations without displaying a user interface.The system may complete remediations without any user or admininvolvement at all. In some embodiments, the system programmaticallydetects an error with about 100% certainty and fixes the issue withabout 100% certainty. In some embodiments, the system may makeremediations without negatively impacting any other elements that mightbe on the page/view. In some embodiments, the manual remediations can beconfigured by developers writing JavaScript, which may be configured inthe Admin (snippets of JavaScript). In some embodiments, the manualremediations may be published via the JavaScript payload delivered tothe client sites that have embedded the script, fixing issues ofaccessibility as programmatically configured/authored by authorizedusers of the platform.

The system may suitably implement individually, or in combination,manual, GUI-based, and automated remediations by acquiring remediationcode during the AE JavaScript loading process, and apply appropriateremediation code to the rendered content. Remediation code may beacquired as part of the AE JavaScript's initial payload, from a specificURL, or from the AE hosting infrastructure.

In various embodiments, remediations immediately manipulate the DOM,causing the corrections and enhancements to take effect at the time thecontent has finished loading and rendering. In some embodiments,keyboard users may be able to use standard keys, such as tab or anyother key or combination of keys, to navigate through the remediatedpage, regardless of whether a screen reader or other AT is beingutilized, reaping the benefits of the applied DOM manipulations in realtime. For users that do not use a stand-alone or native screen reader,the AE JavaScript can comprise a player function which may be configuredto read the remediated page aloud, as described in greater detail below.

FIGS. 1-3 together illustrate a remediation environment in accordancewith various embodiments. More particularly, FIG. 1 is a schematic flowdiagram of an example embodiment of a crawler or “spider” process 100capable of generating a list of URLs to be included as part of aninitial, ongoing, or point-in-time compliance assessment or scan of awebsite. In particular, the process 100 may utilize a configuration file(stored locally or received from an external source) to determine thedepth of the scan, typically expressed in terms of page depth or levelsof recursion, the maximum run time, and the starting URL. The startingURL can be added as the first element to a page stack or memory 110.Other advanced configuration options may be available, allowing forconfiguration of a User Agent, browser dimensions, starting page, anoption to include subdomains in the scope, cookies to include inrequests, usage of a defined Site Map in place of spidering, amongothers.

With continued reference to FIG. 1, in some embodiments, when the AEJavaScript commences the initial scanning phase, the system may connectto a bridge 104 which spawns a headless browser session (i.e.,PhantomJS, Selenium Grid, Google Chrome, Mozilla FireFox) 106.Successive URLs can be retrieved and rendered by a link extractionprocess 108 which may detect some or all links within the rendered DOMand may add unique URLs to the stack 110. The process 100 may retrievesuccessive URLs from the stack and continue crawling until the site isfully cataloged, timeout is reached, or page depth has been exhausted.In the event that the number of pages returned by the spidering processis below a configurable threshold, alternate failover techniques can beutilized to obtain the list of pages, including utilizing Internetsearch engines to request some or all indexed pages matching the spiderconfiguration parameters. In some embodiments, if less than apredetermined number of pages are returned, the system can use a webtool such as cURL to request “searchprovider.com/q=site:xyz.com”. Googlecan return a first results page with a first number of pages in itsindex for the requested domain. The system can store the unique URLsreturned on the results page and run cURL again, requesting one or moreadditional results pages. The system can continue iterating through thisloop until there are no more results pages, or until the system has gonethrough two or more results pages without adding any additional uniqueURLs to the list.

In some embodiments, after the crawler or “spider” process is complete,the scanning process initiates. In some embodiments, the scanningprocess ingests a set of pages, which may be manually configured,attained, for example, via the foregoing crawl/spider process or avariant thereof, or from previous scans. In some embodiments, onceingested, the scanning process can iterate through each unique URL,loading the page in a headless browser, applying the AE JavaScript,executing a series of tests, and ultimately delivering the results ofthe scan back to another process, system, or storage medium.

In some embodiments, once the website is scanned and initial fixesimplemented (e.g. through the inclusion of the AE JavaScript), sitedevelopers may access a development portal and attend to manual fixes,as discussed in greater detail below. Concurrently, users may access theremediated site and enjoy the resulting enhanced user experience evenbefore manual remediation is complete.

Referring now to FIG. 2, a combined schematic block and schematic flowsequence diagram 200 illustrates an exemplary apparatus and method forrendering a web page or other AE JavaScript enhanced user interface inaccordance with various embodiments herein.

More particularly, the system may comprise a user device 202 suitablyequipped with a web browser 204, a first server 206 configured to host awebsite 210 enhanced with an integrated AE JavaScript code base 208, anda second server 212 configured to host various processes and servicesaccessible by the first server including, for example, a RADAE oranalogously functional module 214 and a TTS module 216. It will beunderstood that the use of the phrase “server” in conjunction with items206 and 212 is not intended to be limiting, and that this entity mayinclude a wide range of infrastructure components, such as multipleservers of various types, load balancers, routers, switches, and thelike.

In accordance with various embodiments, the browser 204 may request aweb page (218) from the first server 206, whereupon the AE JavaScript208, executing in the browser 204, retrieves the RADAE module (220) fromthe second server 212 and returns the RADAE module to the user device202 for execution by the web browser 204 (222). The first server mayalso send the HTML page to the browser (224). The browser may requestany additional resources (226), whereupon the AE JavaScript may returnremediation code (228) and the page can be rendered. In someembodiments, once rendered, the remediation code may execute,transforming the DOM into an accessible state. In some embodiments,transforming the DOM may comprise remediating elements by tagging themwith appropriate labels and attributes that may help facilitatecommunication with Assistive Technologies, such as screen readers. Insome embodiments, the system may also dynamically insert off screenmessages and ARIA labels and other best practices that are intended forconsumption by the Screen Reader.

With continued reference to FIG. 2, in some embodiments, when thebrowser requests page content (230), such as a text-to-audio conversion,the AE JavaScript 208 running in the local browser 204 may request audiofrom the second server 212 (232), and the audio may be passed directlyback to the browser 204 for playback to the user (234). In someembodiments, the AE JavaScript facilitates a modular implementation ofvarious ATs, including, for example, an audio player function, a readertool suite (e.g., font face/size, magnification, contrast, line/letterspacing), a voice command feature, a normalized keyboard-friendly sitemenu, and a page elements menu.

FIG. 3 is a schematic flow diagram illustrating an exemplary remediationdelivery process. In some embodiments, once the AE JavaScript loads(302), manual remediations present in the configuration can be appliedand executed (304). A scanning and detection system can scan the page todetect compliance issues (306). Using, for example, a heuristics engine,detected compliance issues can be mapped (308) to pre-existingremediation scripts 312 and programmatic remediation can beexecuted/applied (310) for each mapped compliance issue.

FIGS. 4 and 5 are schematic flow diagrams illustrating creation anddelivery of form-based or “quick-fix” remediations in accordance withvarious embodiments. In accordance with these techniques, remediationsthat would otherwise require coding techniques can be created anddeployed utilizing a simplified dynamic user interface. As used herein,the phrases “quick-fix” and “quick-fix code” refer to code (e.g.,JavaScript code) that, with the addition of some metadata to be providedby an admin, can be used to generate actual remediation code.

More particularly, referring to FIG. 4, an exemplary process may beginwith an admin (or other user authorized to create remediation code)browsing a dynamic user interface (UI) (402), and identifying one ormore elements for which quick fixes are potentially applicable (404).Example quick-fixes include, without limitation, missing or inadequateALT-text, link images with missing or inadequate ALT-text, input imageswith missing or inadequate ALT-text, iFrames missing titles, headers outof order (heading level issues), inputs missing labels, and linksmissing labels.

The admin or authorized user may populate a web-based form (406) thatcan collect data (e.g., metadata) that may be required for performingthe remediation, including the context in which the fix is to beapplied. Such context might include, for example, the scope of therequired remediations in terms of instances and pages (e.g., thisinstance alone, all instances of this issue on one page, all instancesof this issue on every page, etc.).

In some embodiments, the admin or authorized user may submit theweb-based form to the server infrastructure (408), whereupon theremediation code 410 can be programmatically generated and stored. Insome embodiments, all quick-fix remediations of the same type (e.g.,image ALTs) can be stored together in the same remediation storagelocation for a given website.

Referring now to FIG. 5, in some embodiments, after the creation andstorage of quick-fixes, an end-user 502 may browse a web site 504 thathas the quick-fix delivery code embedded therein (e.g., via the AEJavascript). The delivery code can request applicable fixes from serverinfrastructure 506, which may pull the appropriate quick-fix remediationsets 508 from database 510 (or via an appropriate caching mechanism),and may deliver the code to the browser being used to access website504. In some embodiments, once code 508 is received by the browser, itcan be executed, performing the necessary DOM modifications createdpreviously, for example, via the process described above in conjunctionwith FIG. 4.

In this regard, while the quick-fix remediation process is described inthe context of a traditional web form, the various embodiments are notlimited. A variety of other user interfaces may be utilized in a waythat obviates the need for traditional coding processes, which oftenrequire the creation of text documents that are subsequently compiled,interpreted, or converted to an intermediate form of code. Statedanother way, in the context of the AE JavaScript, the admin need not befamiliar with the particular syntax and semantics of programminglanguages such as JavaScript.

FIG. 6 is a schematic flow diagram illustrating an exemplary remediationprocess incorporating a machine learning system in accordance withvarious embodiments. More particularly, when provided with a set ofremediations generated in any suitable fashion (e.g., manually,programmatically, and/or via the quick-fix paradigm described above), amachine learning system in accordance with various embodiments may beconfigured to learn from past remediation history, allowing it to createand suggest future fixes based on past experience.

In accordance with various embodiments, machine learning system 604 mayidentify and analyze a set of issues (608) associated with a set ofanalyzed web page elements (604) from one or more web pages 602. In someembodiments, the system can compare the identified issues withpreviously identified and resolved issues, and ascertain a previouslyapplied remediation that may be applicable to the current issue (610)from one or more issue and remediations databases 612. In other words,the system may identify remediation code created previously that wouldlikely resolve a current issue. Machine learning system 604 may then beable to recall, and possibly modify, existing remediations (614) suchthat they are tailored to address the specific instance identified bythe scanning system.

In some embodiments, once the machine learning system 604 has identifieda remediation (610) that is applicable, the system may store thesuggested remediation code (616) in a suggested remediation codedatabase 618. The admin or authorized user may then be able to browsethrough identified issues using a suitable user interface (e.g.,web-based graphical user interface), select an individual issue, andview the suggested remediation code. The admin may then reject theremediation code, accept the code as presented, or modify the codemanually to address the identified issue (620).

In some embodiments, as training of machine learning system 604continues, it may utilize data gathered from the process to determine aconfidence level associated with each remediation that it generates orsuggests. For example, the system 604 may determine with 96% confidence,based on, for example, past results, that a particular image requires amodified ALT text. In some embodiments, once the confidence level isgreater than a predetermined threshold (e.g., about 99%, about 98%,about 97%, about 96%, about 95%, about 90%, about 85%, about 80%, about75%, about 70%, etc.), the system may be configured to automaticallyapply remediations 604 without requiring that an admin approve eachinstance. In some embodiments, the admin or authorized user has theability to review automatically applied remediations and modify ordisable them as deemed appropriate.

Machine learning system 604 may be implemented in a variety of ways, andmay be trained using supervised, unsupervised, semi-supervised, orreinforcement learning paradigms. Machine learning system 604 mayperform classification (e.g., binary or multiclass classification),regression, clustering, dimensionality reduction, and/or other suchtasks. Examples of suitable machine learning models include, withoutlimitation, artificial neural networks (ANN), decision tree models (suchas classification and regression trees (CART)), ensemble learning models(such as boosting, bootstrapped aggregation, gradient boosting machines,and random forests), Bayesian network models (e.g., naive Bayes),principal component analysis (PCA), support vector machines (SVM),clustering models (such as K-nearest-neighbor, K-means, expectationmaximization, hierarchical clustering, etc.), and/or linear discriminantanalysis models.

In accordance with some embodiments, common templates or structuralpatterns can be identified across multiple pages of a website, allowingthe system to determine a set of remediations that can be applied tomultiple pages without incurring the time and expense of separatelyanalyzing and determining fixes for each of the pages separately. Thisidentification of templates may be performed by any of the systemsdescribed previously, such as server 206 of FIG. 2.

Templates may be identified in a number of ways—e.g., by one or more of,identifying patterns in DOM structure, identifying patterns in URLstructure, and identifying patterns in accessibility test results. Moreparticularly, FIG. 7 is a schematic block diagram illustrating templateidentification utilizing a DOM structure in accordance with variousembodiments. That is, consider a DOM 700 corresponding to a particularexample web page having a variety of elements (e.g., 702, 704, 706, 708,710, 712, 714, 716). As illustrated, the elements may exhibit aparticular hierarchy or topological structure, which might correspond tocommonalities in placement, class, and appearance of web page elements.These similarities in structure and content may then be identifiedacross multiple pages of a website. For example, multiple pages mightcontain a single main navigation block located in the same place andcontaining the same content, a single footer region located roughly inthe same place in the code and containing the same content, and a maincontent area that may exist in the same place within the code for allpages but may contain different contents on each page. In such cases,the system might determine that a common template (and therefore acommon set of remediations) can be applied to all of the pages fittingthat template.

FIG. 8 illustrates an exemplary common template identification schemeutilizing structured URLs in accordance with various embodiments. Thatis, FIG. 8 lists an exemplary set of three website URLs eachcharacterized by a path structure in which a path name 802 (e.g.,“products”) occurs after the domain name, followed by a set of pathnames 804 that begin with the string “model” and terminate with analphanumeric string that varies from page to page (e.g., “11111”,“2222”, “3333”, etc.). That is, the URLs might be identified by thesystem as two instances of the same product detail page. In such a case,the system may identify that the pages exhibiting the illustrated URLpattern correspond to a common template. Those skilled in the art willrecognize that this type of URL inspection can apply to additionalschemes, including patterns in query strings.

In accordance with another embodiment, patterns in accessibility testresults may also be used to identify common templates. That is, usingthe data from the scanning and/or monitoring systems (as describedbelow), it is possible to cross-reference test successes and failures inorder to group web pages together as part of a template.

For example, multiple web pages may include a group of elements thatpass one subset of tests, and another group of elements that failanother subset of tests (with, perhaps, varying results for somespecific content area of the page). In such cases, the system mightconclude that the web pages conform to the same template, with uniquecontent contained within the non-matching areas or elements.

FIG. 9 is a schematic flow diagram illustrating an exemplary monitoringsystem in accordance with various embodiments. That is, a website 902may comprise tracking code that may activate an event (e.g., ananalytics event) 904 upon each page load by a browser, with one or morepage load events being stored within analytics database engine 906.Event 904 includes, in some embodiments, the full page URL and the dateand time of the request.

In some embodiments, monitoring system 908 may request some or allunique URLs from analytics database engine 906, subject to somefiltering criteria. For example, monitoring system 908 may request someor all URLs and the number of times each URL was requested during agiven data range. Monitoring system 908 may send the requested data toaccessibility scanning system (or simply “scanning system”) 910, whichcan load (e.g., using a headless browser) each web page and perform anumber of accessibility tests. Scanning system 910 can store, in testdatabase engine 912, information relating to the tests—e.g., the fullURL, the tests passed by each element, the tests failed by each element,and aggregate counts of test passes and failures. In some embodiments,the database engine used to store results may be a high-performanceanalytical engine such as Amazon RedShift, enabling big-data analyses tobe performed against a large data set, supporting efforts of targetingcommonalities within the data set that can be used to prioritizeexisting issues of accessibility.

Monitoring system 908 can extract data (e.g., metadata) from the testdatabase engine 912 and/or other data sources and can use that data togenerate reports and user interfaces that can be used to prioritizeissues by their impact. In this regard, the “impact” of an accessibilityissue may be characterized by any appropriate metric, including, forexample, the severity of the issue, the number of web pages on which theissue occurs, the total number of effected pages, the number of timeseach page has been accessed, and the like. In some embodiments,monitoring system 908 identifies elements with issues of accessibilityand counts the pages in which that element exists. The system thenutilizes the traffic data—e.g., the number of times each page wasserved—to show the admin a list of elements that, if remediated, wouldoptimally impact overall accessibility and compliance.

FIG. 10 illustrates one non-limiting example of an output table 1000that might be generated by, for example, the monitoring system 908 ofFIG. 9. In some embodiments, output table 1000 includes an “elementreference” field, an “issue detected” field, an “occurrence count”field, a “pages” field, and an “example page” field. The “elementreferences” field may include a suitable identifier or reference to theelement in the DOM. The “issue detected” field may provide a summary ofthe accessibility issue. The “occurrences” field may indicate, forexample the number of times that the subject issue has occurred (whichmay be a function of how many times a page was accessed). The “pages”field may indicate, for example, the number of pages (e.g., within thewebsite as a whole), that the issue occurred. The “example page” fieldmay describe, for admin convenience, an example page in which the issuehas been detected. It will be appreciated that the table illustrated inFIG. 10 is not intended to be limiting, and that any suitable graphicaland/or textual representation may be employed.

FIG. 11 is a schematic flow diagram illustrating an automatic scanfrequency adjustment system 1100 in accordance with various embodiments.In some embodiments, monitoring system 1102 (which may be implemented asdescribed above in conjunction with FIG. 9) may automatically determinehow frequently a given web page should be scanned for accessibilityissues, and may update its scheduling data in a database 1104 or otherdata source accordingly. In some embodiments, an admin or authorizeduser may be given oversight with respect to system 1102 and may beprovided a way to approve any change request relating to the scanning.

Tracking and/or estimating how frequently a page changes (or may changein the future) can be accomplished in a variety of ways. In someembodiments, this tracking includes monitoring a combination of datapoints, such as HTTP header metadata (e.g., content-control orlast-updated), comparing scanned page HTML (or a stored hash) tohistorical versions of same, comparing changes in failure conditions,such as total number of identified errors and risks, as well as fixederrors and risks, and changes in access patterns.

In some embodiments, monitoring system 1102 may request page accessactivity from analytics engine 1106 and/or scheduling database 1104. Foreach distinct web page 1108 returned (which may include a tracking codefor analytics), monitoring system 1102 may load the web page (e.g., in aheadless browser 1110) and execute various accessibility tests 1112. Theoutput of accessibility tests 1112 can be stored in database 1104, andmay include such items as, for example, test results, analytics datainitially received, page HTML, a hash of the HTML, and any otherrelevant data.

As an example, a particular web page 1108 may include a content-controlheader that indicates that the page changes daily; however, themonitoring system may determine that, since the hash of its page contenthas not changed in at least seven days, the page has an update frequencyof not more than weekly. In a further example, a header may indicatethat a page changes monthly; however, the system may determine that thenumber of errors identified on that page changes on a more frequentbasis, and as such the page may be scanned daily until a more accurateand stable estimate can be determined.

FIG. 12 is a schematic flow diagram illustrating the use of a personalinformation store via voice-based interactions in accordance withvarious embodiments. In general, the personal information store mayallow a user to utilize a user profile and voice-based interactions tostore and recall information, for example, form fill-out data.

Referring to FIG. 12, the user, through an access point 1202, can createa profile to be stored a user profile database 1204 using either apersonal identifier such as, for example, an e-mail address, or athird-party single sign-on (SSO) provider such as, for example, Facebookor Google. In some embodiments, once the user is authenticated by thesystem, the user may be able to use voice commands to store and recallinformation.

In some embodiments, the user may visit a website 1206 that may includevoice code from a voice platform 1208, and may initiate voice functionsvia, for example, keyboard or mouse actions. In some embodiments, aftercompleting a login process through a login module 1210 to authenticatewith platform 1208, the authenticated user may issue a voice command1212 instructing the system to store, for example, his or her homeaddress. This may be accomplished, for example, via a speech utteranceof the form, “remember my home address is 123 Maple Street, Tucson,Ariz., 85711.”

The voice command 1212 can be streamed or otherwise transmitted toplatform 1208, where it can undergo local automated speech recognition(ASR) or can be passed off to another internal or external ASR system1214 for ASR processing. The resulting information can be stored in aASR database 1216.

Other examples of information that might be stored and recalled viavoice commands include, for example, phone numbers, banking information,website addresses, password manager information, payment card numbersand expiration dates, email addresses, salutations, social securitynumbers, and/or any other alphanumeric string information.

In some embodiments, using web browser technologies and scriptingtechniques, a voice interaction may be triggered to authenticate a user,permitting access into a secure, password protected environment. Assuch, a voice command, which may be unique to the individual user, canbe spoken and the resulting voice file can be processed by a server tomatch the recording with a globally and securely stored audio file, assupported by or in conjunction with, for example, Single Sign OnTechnologies and Technology Providers. Through, for example, a complex,multi-step process or a more pragmatic, heuristic process (e.g. built-inor leveraging secure 3rd party API integrations), the two files can becompared and contrasted. In some embodiments, when a match isprogrammatically determined, the user can be authenticated permittingaccess to the secured environment.

In some embodiments, when a user visits a web page that includes inputsfor addresses or the like, the user may issue a voice command 1212, suchas “fill form with my home address” or “insert cellular phone number.”The voice code can request the home address data or other applicableinformation from platform 1208, which can fill the appropriate field orfields with the stored data.

The above embodiments may be employed with web form inputs, such as, forexample, address, contact info, payment info, and the like, but may alsoaccept free-form information for storage, recollection, andtransmission. For instance, a user may configure the system to store hiswork email address, and then utilize that information for datatransmission. In some examples, the user may issue a voice command 1212to store some vocalized information as “note one.” The user can thenissue a voice command to “send note one to my work email,” causingplatform 1208 to generate an email to the user's work email address,with the contents of the message body being the contents of “note one.”Similarly, an authenticated user may utilize the user interface providedby the voice code to populate user-defined data fields, i.e., “JohnDoe's Christmas List.”

In accordance with another embodiment, the user may store custom“overrides” in the user profile database 1204. For example, the user mayissue the command, “when I say Jeff, enter Jeffrey.” In someembodiments, after receiving confirmation that the platform 1208 hasstored the personalization override, each future occurrence of the userrequesting “Mike” can be replaced by the string “Michael.”

FIG. 13 is a schematic block diagram illustrating an example embodimentof a platform for handling “series commands,” which may allow a user toissue a single contiguous voice command that may cause the system toexecute multiple interactions on behalf of the user. The interactionsmay emulate the exact actions that a user would take (e.g., moving thecursor, clicking a link, etc.), or they can execute functions directly,depending upon various factors.

FIG. 13 illustrates an example in which a user wishes to view her latestpay stub via a website that includes a long series of pages leading tothe desired information. That is, a user(s), through a user access point1302, can begin their interaction on a web page 1304, speaking a voicecommand 1306 such as “view my latest pay stub.” Voice platform 1308(optionally including a voice module) may receive voice command 1306 andconvert it to an intent using a combination of one or more ASR and NLPsystems 1310. The intent or text (represented using any convenient datastructure) can be returned to voice system 1308, which can retrieve(e.g., from intent database 1312) the steps necessary to execute theintent.

In some embodiments, the steps can be formatted as JavaScript code,structured data (e.g., JSON data), or any other such code that canrepresent the required steps. The program code can be delivered back touser access point 1302 (i.e., the user's browser) for execution.

In some embodiments, the intent can engage the “pay page” link 1314 onpage 1304. The code can execute a click event on the link 1314 and theuser's browser can navigate to the “pay page” 1316. In some embodiments,the voice code can determine that it is currently engaged in a seriescommand, and can identify that the next step in the process may be tofind and click the pay stubs link 1318 to reach the pay stubs page 1320.The check stubs link 1322 can then programmatically clicked, and thebrowser can navigate to the check stubs page 1324. In some embodiments,the system may identify the current state of the series commandexecution and iterate engaging one or more links 1314, 1318, and 1322 todisplay the latest check stub 1324.

In accordance with various embodiments, a modified version of theprocess described above in connection with FIG. 13 can be employed toprovide a guided tutorial. That is, rather than automatically executingthe series of commands by programmatically (or through emulation)clicking the correct series of links or otherwise executing the steps ofthe intent across multiple pages or within multiple views within aSingle Page Application (SPA), the user can be provided with a visualindicator or other assistive technology to indicate which links shouldbe clicked and in which order. In some embodiments, once the user hasnavigated (manually) to the target page (e.g. check stubs page 1324),the series command may be complete and execution may end.

FIG. 14 is a schematic flow diagram illustrating a system fordynamically routing voice commands to an appropriate, optimal, best-fit,or otherwise desirable natural language processing (NLP) and/orautomatic speech recognition (ASR) system in accordance with variousembodiments.

In some embodiments, when converting spoken voice input into intents,each available ASR and NLP engine may be more suited to particularcontexts and applications and less suited to others. For example, thereare ASR systems that excel at understanding proper nouns, while thereare others that are better suited for dictionary terms. Likewise, thereare NLP engines with high levels of accuracy with respect tounderstanding spoken food items, while others are better atunderstanding mathematical terminology.

In some embodiments, when a spoken utterance is received by the voicesystem, it may be the case that contextual clues can provide enoughinformation to route the utterance to an appropriate combination of ASRand/or NLP engines. In other embodiments, no contextual clues may beapparent as to what type of information is being spoken by the user. Insuch embodiments, a sequential processing workflow may be required inwhich the audio is fed first into one processor, then into another, withthe results being compared to determine next steps.

For example, a user may issue the command, “I need to take off the nextthree days for vacation” in conjunction with a web page form thataccepts start date, end date, and pay code for submitting time-offrequests. In the event NLP rules can be created to handle thisfunctionality, the voice system might route the audio first to theconfigured ASR, and then send the returned copy through a selected NLPengine. In some embodiments, some NLP engines contain ASR functionality,so the illustrated workflow may provide the ability to make decisions asto the appropriate NLP and ASR engines for a given scenario. In someembodiments, once the NLP engine has returned the appropriate intent andvariables (e.g. converting next three days to the current date, and thedate three days from now), the code may execute on the page to make theappropriate selections on the form.

Referring to FIG. 14, the task of dynamically routing voice commands maytake place as follows. In some embodiments, a user, through a useraccess point 1402, may access web page 1404, which may include a voicemodule 1406. The user may issue a voice command 1408 to voice platform1410 via voice module 1406, which may utilize information contained onweb page 1404 or from a context database 1412 to determine context andan appropriate routing path.

In some embodiments, voice platform 1410 may be communicatively coupledto a plurality of ASRs 1414, and a plurality of NLPs 1416. In someembodiments, a route can be configured and retrieved from a database1412, which can instruct the voice platform 1410 to send the commandfirst to one or more ASRs of the plurality of ASRs 1414, and can sendthat output to one or more NLPs of the plurality of NLPs 1416 for intentprocessing. The results from one or more NLPs of the plurality of NLPs1416 can be sent back to voice platform 1410 for processing. The voiceplatform 1410 can send the appropriate program code back to voice module1406, where it can be executed and the intended action can be performed.

In cases where no pre-defined logic or path exists, the voice platform1410 may be configured to first send the audio to one or more NLPs ofthe plurality of NLPs 1416, and if no fully formed user action isdetected, send the same command to one or more different NLPs of theplurality of NLPs 1416. If no fully formed user action is detected, thesystem may be configured to default to internal rules for subsequent orfinal action determination. In some embodiments, the system receivescommands that are not properly married to an intent, and in such casesthe system dynamically pairs the command to an intent, or a systemadministrator, through, e.g., a GUI, maps the command to an intent. Insome embodiments, when the same command is detected again, the intentand associated programmatic steps can be carried out.

In accordance with some embodiments, the user may be required to fill inany logical gaps, i.e., by engaging in an interactive dialog with voiceplatform 1410. For example, if a user issues the command “scroll,” thesystem may prompt the user (visually or via an audio or text prompt) fora particular direction (e.g., “up”, “down”, etc.). In some embodiments,logical gaps are not limited to single data points. For example, if therequested action is to process a time-off form, and the required datapoints are start date, end date, and pay code, a user issuing the timeoff command, providing only the pay code, may be promoted to respondwith both the start date and end date.

FIG. 15 is a flowchart illustrating an exemplary method for dynamicallyrouting voice commands in accordance with various embodiments. In someembodiments, the method may include receiving a voice command (1502). Insome embodiments, at 1504, the system may determine contextualinformation (e.g., from the current website, a database, etc.). Based onthe contextual information, the system may select a recognition pathincluding a first ASR or NLP to send the voice command (e.g., an audiorepresentation of the voice command) to the selected ASR or NLP (1506).At 1508, the system may receive the output from the selected ASR or NLPand, based on that output, may alter the recognition path to send theoutput (and/or the original voice command) to a second ASR or NLP(1510).

FIG. 16 illustrates an exemplary user interface including numericallylabeled elements in accordance with various embodiment. Moreparticularly, referring now to FIG. 16, an exemplary web page 1600(which might correspond to any of the user interfaces described herein)may include a plurality of interactive elements 1602, 1604, 1606, and1608. Such interactive elements might include, for example, hyperlinks,buttons, drop-down menus, radio buttons, or any other user interfaceelement. As shown in FIG. 16, each of the interactive elements 1602,1604, 1606, and 1608 can be accompanied by a corresponding, uniquenumeral code, alphabetical code, alphanumerical code, shape, and/or anyother representative code (in this example, the numerals 1-4,respectively) that is displayed, for example, via the AE JavaScript. Thenumerals or any other representative code may be any convenient size,font, etc., and may be positioned above, to the side, or otherwiseadjacent to their respective interactive elements. In accordance withvarious embodiments, the system may receive a voice command from theuser corresponding to one of the displayed numerals or any otherrepresentative code, and can activate the corresponding interact element(e.g., by programmatically clicking the selected element). In someembodiments, for example, the user may say “go to three” or simply“three,” prompting the system to activate button 1606.

FIG. 17 is a schematic flow diagram illustrating the use of a middletier in connection with a voice device (e.g., Google Home, Amazon Echo,etc.) in accordance with various embodiments. In some embodiments, itcan be assumed that an action database 1702 has been previouslypopulated with information associated with the performance of arequested action, the requested action being associated with aparticular service provider (e.g., a shopping site, a restaurantproviding online ordering, or the like). Such information may include afavorite set of items or other preferences that the user had previouslyprovided in connection with accessing that service provider's website.

With continued reference to FIG. 17, user 1704 may issue a voice command1706 that can be processed by a voice device 1708. In some embodiments,voice device 1706 may utilize an installed skill or other residentsoftware code 1710 when it sends the request to a server 1712 (e.g. aserver that is used to provide the AE JavaScript). Server 1712 mayretrieve relevant information from action database 1702 to perform therequested action via service provider 1714.

FIG. 18 is a schematic flow diagram illustrating an example embodimentof a system 1800 for employing a middle tier in connection with a voicedevice such as that shown in FIG. 17. In some embodiments, no APIs areavailable from a service provider for performing actions outside of anormal web or app interaction. In these cases, skill 1810 may act as themiddle-tier to execute requests on the user's behalf. Specifically,skill 1810 may send a request to server 1812 and server 1812 mayretrieve user information from database 1802. Server 1802 may use aheadless browser 1814 to log into service provider website 1816, and mayperform the action requested by the voice skill.

FIG. 19 is an exemplary web page illustrating the use of a portabledocument format generation tool in accordance with various embodiments.In this regard, the term “PDF” for “portable document format” will beused herein without loss of generality. The methods described herein maybe used in connection with other such formats (e.g., XPS).

In some embodiments, the accessibility of PDF documents may be increasedby adding accessibility tags to the document (i.e., “tagging”). Inaccordance with various embodiments, however, users and admins are ableto create HTML versions of PDF documents while also being able to exporttagged PDF files those having the appropriate reader software.

Referring to FIG. 19, a web page 1900 may comprise a variety ofcomponents that allow a user (or admin) to create an alternate (e.g.HTML) version of a PDF document, as illustrated conceptually in thefigure. Specifically, web page 1900 may comprise a series of buttonswhich might include, for example, an accordion menu 1914 that mayprovide an index or table of contents for the other elements appearingon webpage 1900. In some embodiments, shown along the top of web page1900 may be a set of buttons or other user interface elements 1902,1904, 1906, 1908, 1910, and 1912 that facilitate the creation of, forexample, the HTML document. For example, user interface elements may beassociated with performing optical character recognition (OCR) on aportion of the page (1902), adding a static text element (1904), addingauto-detect rows (1906), adding auto-detect columns (1908), addingauto-detect list items (1910), and adding a custom helper (1912).

In accordance with some embodiments, the web page 1900 may also include,for example, a canvas layer 1914, a PDF overlay 1916, an SVG layer 1918,a sample header slide 1920, a sample initial helper 1922, a static textlayer slice 1924, and a dynamic text layer slice 1926. In someembodiments, using the available controls, the user may identify (withinthe displayed PDF document) each content area of a document (e.g., usinga rectangular selection tool), and may associate the content block withthe most appropriate accessibility tag. For example, a title displayedat the top of the first page of the PDF document might be selected andmarked as the highest level heading (hl).

In accordance with various embodiments, processing may proceed in thefollowing manner. In some embodiments, the system can create an HTML5canvas layer (1914) having the PDF content (1916) rendered within it.The system may extract, when possible, blocks of content from therendered PDF that can be programmatically identified as being of acertain content type, and may automatically generate “initial helpers”for these elements. For example, the system may recognize that there arebullets on the page that are aligned vertically, therefore assuming thiscontent to be an unordered list. In some embodiments, the system couldgenerate an initial helper whose slice would encompass the entire list,with the helper type being set to “list” or similar (e.g., 1922).Initial helpers can be sorted by top and left x/y coordinates to placethem in a default order. These x/y coordinates may be automatically ormanually determined. The user can extract, when possible, blocks of textfrom the rendered PDF content, and these blocks can generate entries inaccordion (1914) in the same way that entries are generated during the“initial helper” process described above. In some embodiments, some orall elements can be appended to a parent page element and an SVG element(1918) may be added on top of the canvas layer to allow rectangles to bedrawn, thereby defining which texts blocks should be considered anindividual entity.

In some embodiments, when a slice (e.g., 1920, 1922, 1924, and 1926) isdrawn (either manually or autonomously) on the SVG layer 1918, the PDFaccessibility tool can retrieve any text from the underlying text-layerelements within the rectangular coordinates. If no text is detected, thesystem may attempt to perform an OCR of the area inside the coordinatesusing one or more computer vision APIs. The helpers can be added, forexample, to an accordion display (1914) within a form that allows theuser to set element types, replace text, and/or change other attributesas appropriate. The element types might include, for example, basic HTMLelements such as paragraphs, headers, unordered lists, and tables, asthose types map roughly to both HTML elements as well as documentaccessibility tags.

In some embodiments, if difficulties are experienced detecting aparticular portion of text, the user may engage the OCR function (1902)to extract some or all text from the current page. The result of the OCRprocess can be merged with the initial text in order to provide moreaccurate text data.

In some embodiments, some or all work performed by the user in creatingthe HTML version of the PDF document can be stored as a separateversion, with changes only being visible to end-users after promotion ofthe document to production level.

In some embodiments, with respect to displaying the PDF content on awebsite, the client can create a blank page on their website thatsubstantially includes only the AE Javascript. On pages that containlinks to PDF files that have been remediated, the user may be given thechoice to download the original PDF or view the accessible version.Selecting the accessible version may link them to the blank page,passing in the URL of the PDF. On this page, a pdf-reader and playercode may load. The code may detect the rendering of the text layer andretrieve the elements in order to manipulate them. The elements can besorted by left and top edges with an adjustment based, for example, onheuristic rules (e.g., to account for bolded text in the middle of asentence, or the like). The array of sorted elements can be used tocreate an array of text lines.

In some embodiments, if helpers as described above exist on the page,the system may loop through them and add them to the array of lines atthe appropriate place (based, for example, on their top position). Usingthe helper item type the system may create the elements to which theHTML is to be applied. The system may attempt to ensure the appropriatedata attributes are added to the elements in order to create the taggedPDF documents.

In some embodiments, the initial text layer may be replaced with thenew, structured HTML content. In some embodiments, as the pages areloaded, the system can store the HTML in an array together with a dataURI containing a representation of the canvas image. In someembodiments, when the download button is clicked, the array can belooped through to retrieve the HTML and the URIs for each page and applyit to a template HTML file with the image from the canvas absolutelypositioned over the text layer for each page. The HTML can be sent to aPDF server to generate a downloadable PDF.

In some embodiments, the PDF server may receive the provided HTML andload it with, for example, a customized version of DOMPDF (an HTML toPDF converter that takes HTML and CSS and converts it to a PDF). Themodification might include, for example, applying data attributes fromthe HTML and translating them into attributes that may be required for atagged PDF. In some embodiments, the modifications comprise, forexample, reading the height and width of the text layer in order tochange the paper size and layout dynamically when rendering the PDF. Thecompleted document can be transmitted to the browser where it can bedownloaded.

The auto-detect columns button 1908 may initiate a process thatcomprises obtaining the coordinates of the containing table and creatingan array with the elements within that area. These elements can besorted by their left position and added to an array of columns createdbased on, for example, the elements' right and left coordinates. If anelement is partially within the boundaries of a column, the system mayadd the element to a column and the column coordinates can be updatedaccordingly. Similarly, the auto-detect rows button 1906 may initiate aprocess comprising obtaining the coordinates of the containing table andcreating an array with some or all elements within that area. Theelements can then be added to an array of rows created based on, forexample, the element's top and bottom coordinates.

In some embodiments, the autodetect list items button 1910 may initiatea process comprising obtaining the coordinates of the containing listand create an array with all the text layer elements within that area.Bullets may also be detected based on, for example, the first characterof each row and whether the left position of that letter is less than 10px from the left position of the parent list.

FIG. 20 is a block diagram depicting an embodiment of a computerhardware system configured to run software for implementing one or moreembodiments of the website remediation and assistive technologiessystems, methods, and devices disclosed herein.

In some embodiments, the systems, processes, and methods describedherein are implemented using a computing system, such as the oneillustrated in FIG. 20. The example computer system 2002 is incommunication with one or more computing systems 2020 and/or one or moredata sources 2022 via one or more networks 2018. While FIG. 20illustrates an embodiment of a computing system 2002, it is recognizedthat the functionality provided for in the components and modules ofcomputer system 2002 may be combined into fewer components and modules,or further separated into additional components and modules.

The computer system 2002 can comprise a remediation and assistivetechnologies module 2014 that carries out the functions, methods, acts,and/or processes described herein. The remediation and assistivetechnologies module 2014 is executed on the computer system 2002 by acentral processing unit 2006 discussed further below.

In general, the word “module,” as used herein, refers to logic embodiedin hardware or firmware or to a collection of software instructions,having entry and exit points. Modules are written in a program language,such as JAVA, C or C++, PYPHON or the like. Software modules may becompiled or linked into an executable program, installed in a dynamiclink library, or may be written in an interpreted language such asBASIC, PERL, LUA, or Python. Software modules may be called from othermodules or from themselves, and/or may be invoked in response todetected events or interruptions. Modules implemented in hardwareinclude connected logic units such as gates and flip-flops, and/or mayinclude programmable units, such as programmable gate arrays orprocessors.

Generally, the modules described herein refer to logical modules thatmay be combined with other modules or divided into sub-modules despitetheir physical organization or storage. The modules are executed by oneor more computing systems and may be stored on or within any suitablecomputer readable medium or implemented in-whole or in-part withinspecial designed hardware or firmware. Not all calculations, analysis,and/or optimization require the use of computer systems, though any ofthe above-described methods, calculations, processes, or analyses may befacilitated through the use of computers. Further, in some embodiments,process blocks described herein may be altered, rearranged, combined,and/or omitted.

The computer system 2002 includes one or more processing units (CPU)2006, which may comprise a microprocessor. The computer system 2002further includes a physical memory 2010, such as random access memory(RAM) for temporary storage of information, a read only memory (ROM) forpermanent storage of information, and a mass storage device 2004, suchas a backing store, hard drive, rotating magnetic disks, solid statedisks (SSD), flash memory, phase-change memory (PCM), 3D XPoint memory,diskette, or optical media storage device. Alternatively, the massstorage device may be implemented in an array of servers. Typically, thecomponents of the computer system 2002 are connected to the computerusing a standards-based bus system. The bus system can be implementedusing various protocols, such as Peripheral Component Interconnect(PCI), Micro Channel, SCSI, Industrial Standard Architecture (ISA) andExtended ISA (EISA) architectures.

The computer system 2002 includes one or more input/output (I/O) devicesand interfaces 2012, such as a keyboard, mouse, touch pad, and printer.The I/O devices and interfaces 2012 can include one or more displaydevices, such as a monitor, that allows the visual presentation of datato a user. More particularly, a display device provides for thepresentation of GUIs as application software data, and multi-mediapresentations, for example. The I/O devices and interfaces 2012 can alsoprovide a communications interface to various external devices. Thecomputer system 2002 may comprise one or more multi-media devices 2008,such as speakers, video cards, graphics accelerators, and microphones,for example.

The computer system 2002 may run on a variety of computing devices, suchas a server, a Windows server, a Structure Query Language server, a UnixServer, a personal computer, a laptop computer, and so forth. In otherembodiments, the computer system 2002 may run on a cluster computersystem, a mainframe computer system and/or other computing systemsuitable for controlling and/or communicating with large databases,performing high volume transaction processing, and generating reportsfrom large databases. The computing system 2002 is generally controlledand coordinated by an operating system software, such as z/OS, Windows,Linux, UNIX, BSD, SunOS, Solaris, MacOS, or other compatible operatingsystems, including proprietary operating systems. Operating systemscontrol and schedule computer processes for execution, perform memorymanagement, provide file system, networking, and I/O services, andprovide a user interface, such as a graphical user interface (GUI),among other things.

The computer system 2002 illustrated in FIG. 20 is coupled to a network2018, such as a LAN, WAN, or the Internet via a communication link 2016(wired, wireless, or a combination thereof). Network 2018 communicateswith various computing devices and/or other electronic devices. Network2018 is communicating with one or more computing systems 2020 and one ormore data sources 2022. The remediation and assistive technologiesmodule 2014 may access or may be accessed by computing systems 2020and/or data sources 2022 through a web-enabled user access point.Connections may be a direct physical connection, a virtual connection,and other connection type. The web-enabled user access point maycomprise a browser module that uses text, graphics, audio, video, andother media to present data and to allow interaction with data via thenetwork 2018.

Access to the remediation and assistive technologies module 2014 of thecomputer system 2002 by computing systems 2020 and/or by data sources2022 may be through a web-enabled user access point such as thecomputing systems' 2020 or data source's 2022 personal computer,cellular phone, smartphone, laptop, tablet computer, e-reader device,audio player, or other device capable of connecting to the network 2018.Such a device may have a browser module that is implemented as a modulethat uses text, graphics, audio, video, and other media to present dataand to allow interaction with data via the network 2018.

The output module may be implemented as a combination of an all-pointsaddressable display such as a cathode ray tube (CRT), a liquid crystaldisplay (LCD), a plasma display, or other types and/or combinations ofdisplays. The output module may be implemented to communicate with inputdevices 2012 and they also include software with the appropriateinterfaces which allow a user to access data through the use of stylizedscreen elements, such as menus, windows, dialogue boxes, tool bars, andcontrols (for example, radio buttons, check boxes, sliding scales, andso forth). Furthermore, the output module may communicate with a set ofinput and output devices to receive signals from the user.

The input device(s) may comprise a keyboard, roller ball, pen andstylus, mouse, trackball, voice recognition system, or pre-designatedswitches or buttons. The output device(s) may comprise a speaker, adisplay screen, a printer, or a voice synthesizer. In addition a touchscreen may act as a hybrid input/output device. In another embodiment, auser may interact with the system more directly such as through a systemterminal connected to the score generator without communications overthe Internet, a WAN, or LAN, or similar network.

In some embodiments, the system 2002 may comprise a physical or logicalconnection established between a remote microprocessor and a mainframehost computer for the express purpose of uploading, downloading, orviewing interactive data and databases on-line in real time. The remotemicroprocessor may be operated by an entity operating the computersystem 2002, including the client server systems or the main serversystem, an/or may be operated by one or more of the data sources 2022and/or one or more of the computing systems 2020. In some embodiments,terminal emulation software may be used on the microprocessor forparticipating in the micro-mainframe link.

In some embodiments, computing systems 2020 who are internal to anentity operating the computer system 2002 may access the remediation andassistive technologies module 2014 internally as an application orprocess run by the CPU 2006.

In some embodiments, one or more features of the systems, methods, anddevices described herein can utilize a URL and/or cookies, for examplefor storing and/or transmitting data or user information. A UniformResource Locator (URL) can include a web address and/or a reference to aweb resource that is stored on a database and/or a server. The URL canspecify the location of the resource on a computer and/or a computernetwork. The URL can include a mechanism to retrieve the networkresource. The source of the network resource can receive a URL, identifythe location of the web resource, and transmit the web resource back tothe requestor. A URL can be converted to an IP address, and a DomainName System (DNS) can look up the URL and its corresponding IP address.URLs can be references to web pages, file transfers, emails, databaseaccesses, and other applications. The URLs can include a sequence ofcharacters that identify a path, domain name, a file extension, a hostname, a query, a fragment, scheme, a protocol identifier, a port number,a username, a password, a flag, an object, a resource name and/or thelike. The systems disclosed herein can generate, receive, transmit,apply, parse, serialize, render, and/or perform an action on a URL.

A cookie, also referred to as an HTTP cookie, a web cookie, an internetcookie, and a browser cookie, can include data sent from a websiteand/or stored on a user's computer. This data can be stored by a user'sweb browser while the user is browsing. The cookies can include usefulinformation for websites to remember prior browsing information, such asa shopping cart on an online store, clicking of buttons, logininformation, and/or records of web pages or network resources visited inthe past. Cookies can also include information that the user enters,such as names, addresses, passwords, credit card information, etc.Cookies can also perform computer functions. For example, authenticationcookies can be used by applications (for example, a web browser) toidentify whether the user is already logged in (for example, to a website). The cookie data can be encrypted to provide security for theconsumer. Tracking cookies can be used to compile historical browsinghistories of individuals. Systems disclosed herein can generate and usecookies to access data of an individual. Systems can also generate anduse JSON web tokens to store authenticity information, HTTPauthentication as authentication protocols, IP addresses to tracksession or identity information, URLs, and the like.

The computing system 2002 may include one or more internal and/orexternal data sources (for example, data sources 2022). In someembodiments, one or more of the data repositories and the data sourcesdescribed above may be implemented using a relational database, such asDB2, Sybase, Oracle, CodeBase, and Microsoft® SQL Server as well asother types of databases such as a flat-file database, an entityrelationship database, and object-oriented database, and/or arecord-based database.

The computer system 2002 may also access one or more databases 2022. Thedatabases 2022 may be stored in a database or data repository. Thecomputer system 2002 may access the one or more databases 2022 through anetwork 2018 or may directly access the database or data repositorythrough I/O devices and interfaces 2012. The data repository storing theone or more databases 2022 may reside within the computer system 2002.

FIG. 21 a flowchart illustrating an exemplary system for a voice-enabledkiosk, browser, or point-of-sale user interface in accordance withvarious embodiments herein. In some embodiments, voice-enabled systemsas described herein can be wholly compatible with compliant web sites,web applications, operating systems, user interfaces, browser, andothers through a browser-based controller interface, a secondary device,or other microphone-enabled source. In some embodiments, the remediationsystems, devices, and methods described herein can be used to fixcompliance issues with web sites or user interfaces, such that the websites or user interfaces may be voice-enabled. In some embodiments, thesystem allows full control of the actual functionality and visualpresentation of a browser, website, user interface, or applicationthrough voice command. In some embodiments, common browser functions(e.g. go back, reload, scroll, open settings, go to <insert link namehere>, move to <insert form input name here>, enter <insert form input>)can be activated upon voice command. Additionally, users can createcustomized, application-specific or generalized voice commands accordingto their preferences to perform specific dynamic actions. In someembodiments, the system can act as a voice-controlled hub that directs auser to various other applications. In some embodiments, the system isfully integrated into those various applications to provide a completeand seamless voice-enabled user experience.

Referring to FIG. 21., the system 2100 may comprise a browser, kiosk, orother point-of-sale user interface 2102. In some embodiments, thebrowser, kiosk, or other point-of-sale user interface 2102 may comprisea microphone 2104 and an integrated JavaScript 2106 for performing voicecommand execution. In some embodiments, the web browser, kiosk, or otherpoint-of-sale 2102 user interface may comprise a “click-to-listen”button or it may be configured to detect voice commands automaticallythrough an always listening paradigm. In some embodiments, the end-usermay click a button to prompt the “listening” function. In someembodiments, engagement may be triggered through an always listeningparadigm, whereby actions are triggered by speaking a trigger command,which is fully configurable on a fully custom basis (e.g. “Computer” or<insert company name>). In other embodiments, a voice command can bespoken to a secondary device such as a cellular phone or tablet, whichcan be paired with the browser, kiosk, or other point-of-sale userinterface 2102 to control the system remotely. The pairing process maybe achieved through pairing options such as, for example, QR scanning,secure key-value pair authentication, multi-factor authentication, andsecure link authentication.

In some embodiments, the system may be configured to dynamically routevoice commands to an appropriate, optimal, best-fit, or otherwisedesirable natural language processing (NLP) and/or automatic speechrecognition (ASR) system 2108, as described in more detail above. Insome embodiments, the system will comprise an NPL and/or ASR. Thus, thespeech-to-text conversion and subsequent interpretation can occur withinthe system itself. In other embodiments, the system will transmit anaudio file of the spoken voice command to a third-party NP and/or ASRfor speech processing. In some embodiments, the system can utilize bothan integrated NLP and/or ASR as well as third-party services.

In some embodiments, the output of the NLPs and/or ASRs is directed toan intent/processing engine 2110, which may interpret the intent of thecommand and pair the spoken command into an actionable command. In someembodiments, the actionable command may consist of a single command tiedto a single action. Alternatively, intent may consist of a single orseries of commands that execute multiple operations in series, asdiscussed in more detail herein. Thus, in some embodiments, the systemmay be configured to allow a user to speak a single command thattriggers a series of clicks across multiple pages to, for example, findrequested information or for filling out multiple fields in a web form.For example, the system could be configured to execute a “Check-out”routine, which could trigger a series of steps to facilitate theshopping cart purchase experience within a Single Page Application (SPA)or across a series of pages within a more traditional websiteexperience. In some embodiments, the intent/processing engine 2110 mayutilize one or more command databases (e.g. enterprise database 2112,retail database 2114, and other database 2116), containing contextualdata that can contribution to converting the spoken voice command intoan functional command.

In some embodiments, the system may comprise a custom functionprocessor, which may interpret basic commands tied to common behaviors.This approach bypasses third-party NLP services and adds a level ofspeed and efficiency to the processing required in order to fulfill userintent.

In various embodiments described herein, the system may be intended toprovide a voice-driven user experience that is paired with a visualpresentation layer, i.e. the modern web browser presented from acomputer display or, for example, a kiosk terminal.

Embodiments

Although this invention has been disclosed in the context of certainembodiments and examples, it will be understood by those skilled in theart that the invention extends beyond the specifically disclosedembodiments to other alternative embodiments and/or uses of theinvention and obvious modifications and equivalents thereof. Inaddition, while several variations of the embodiments of the inventionhave been shown and described in detail, other modifications, which arewithin the scope of this invention, will be readily apparent to those ofskill in the art based upon this disclosure. It is also contemplatedthat various combinations or sub-combinations of the specific featuresand aspects of the embodiments may be made and still fall within thescope of the invention. It should be understood that various featuresand aspects of the disclosed embodiments can be combined with, orsubstituted for, one another in order to form varying modes of theembodiments of the disclosed invention. Any methods disclosed hereinneed not be performed in the order recited. Thus, it is intended thatthe scope of the invention herein disclosed should not be limited by theparticular embodiments described above.

As used herein, the word “exemplary” means “serving as an example,instance, or illustration.” Any implementation described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other implementations, nor is it intended to beconstrued as a model that must be literally duplicated.

Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment. Theheadings used herein are for the convenience of the reader only and arenot meant to limit the scope of the inventions or claims.

Further, while the systems, methods, and devices described herein may besusceptible to various modifications and alternative forms, specificexamples thereof have been shown in the drawings and are hereindescribed in detail. It should be understood, however, that theinvention is not to be limited to the particular forms or methodsdisclosed, but, to the contrary, the invention is to cover allmodifications, equivalents, and alternatives falling within the spiritand scope of the various implementations described and the appendedclaims. Further, the disclosure herein of any particular feature,aspect, method, property, characteristic, quality, attribute, element,or the like in connection with an implementation or embodiment can beused in all other implementations or embodiments set forth herein. Anymethods disclosed herein need not be performed in the order recited. Themethods disclosed herein may include certain actions taken by apractitioner; however, the methods can also include any third-partyinstruction of those actions, either expressly or by implication. Theranges disclosed herein also encompass any and all overlap, sub-ranges,and combinations thereof. Language such as “up to,” “at least,” “greaterthan,” “less than,” “between,” and the like includes the number recited.Numbers preceded by a term such as “about” or “approximately” includethe recited numbers and should be interpreted based on the circumstances(e.g., as accurate as reasonably possible under the circumstances, forexample ±5%, ±10%, ±15%, etc.). For example, “about 3.5 mm” includes“3.5 mm.” Phrases preceded by a term such as “substantially” include therecited phrase and should be interpreted based on the circumstances(e.g., as much as reasonably possible under the circumstances). Forexample, “substantially constant” includes “constant.” Unless statedotherwise, all measurements are at standard conditions includingtemperature and pressure.

As used herein, a phrase referring to “at least one of a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: A, B, or C” is intended to cover: A, B, C,A and B, A and C, B and C, and A, B, and C. Conjunctive language such asthe phrase “at least one of X, Y and Z,” unless specifically statedotherwise, is otherwise understood with the context as used in generalto convey that an item, term, etc. may be at least one of X, Y or Z.Thus, such conjunctive language is not generally intended to imply thatcertain embodiments require at least one of X, at least one of Y, and atleast one of Z to each be present.

1. (canceled)
 2. A computer-implemented method of programmaticallyassigning an accessible rich internet applications (ARIA) label to animage on a web page to provide an audible description of the image, theweb page having an associated document object model (DOM), the methodcomprising: detecting one or more compliance issues relating to webaccessibility standards in code associated with a web page, wherein atleast one of the one or more compliance issues comprises an untaggedimage; and mapping the one or more compliance issues to one or morepre-existing remediations, wherein the one or more pre-existingremediations comprise: determining, by a computer system, the subjectmatter of the untagged image using an image recognition algorithm;assigning, by the computer system, an ARIA label to the untagged imagebased on the determined subject matter of the untagged image, the ARIAlabel configured to be used for altering the code of the web page, theARIA label configured to enable an assistive technology speak a messageto the a user, wherein the one or more pre-existing remediations areJavaScript, and storing, by the computer system, the ARIA label in astorage medium.
 3. The computer-implemented method of claim 2, whereinthe computer-implemented method is performed on a periodic basis.
 4. Thecomputer-implemented method of claim 2, wherein the computer systemcomprises one or more computing systems.
 5. The computer-implementedmethod of claim 2, wherein the image recognition algorithm operates on aremote computing system.
 6. The computer-implemented method of claim 2,wherein the untagged image is a video.
 7. The computer-implementedmethod of claim 2, wherein the code associated with the web page is theDOM or HTML code.
 8. The computer-implemented method of claim 2, whereinthe image recognition algorithm is based on using contextual.
 9. Thecomputer-implemented method of claim 2, wherein the ARIA label comprisesa text description of the untagged image.
 10. The computer-implementedmethod of claim 2, wherein the untagged image is an image associatedwith one or more descriptive attributes that is erroneous.
 11. Thecomputer-implemented method of claim 2, wherein the image recognitionalgorithm is based on an artificial intelligence algorithm.
 12. Acomputer-implemented method of programmatically assigning an accessiblerich internet applications (ARIA) label to an image on a web page toprovide an audible description of the image, the web page having anassociated document object model (DOM), the method comprising: detectingone or more compliance issues on a web page relating to webaccessibility standards in the code, wherein at least one of the one ormore compliance issues comprises an untagged element; and applying tothe one or more compliance issues one or more pre-existing remediations,wherein the one or more pre-existing remediations comprise: locating, bya computer system, within the code associated with the web page theuntagged element, the untagged element lacking an adequate descriptiveattribute; assigning, by the computer system, an ARIA label to theuntagged element to provide a descriptive attribute to the untaggedelement, the ARIA label configured to be used for altering the code ofthe web page to enable the ARIA label to be read to a user using anassistive technology, wherein the one or more pre-existing remediationsare JavaScript, and storing, by the computer system, the ARIA label in astorage medium.
 13. The computer-implemented method of claim 12, whereinthe untagged element comprises an input field.
 14. Thecomputer-implemented method of claim 12, wherein the untagged elementcomprises a title element.
 15. The computer-implemented method of claim12, wherein the untagged element comprises an untagged image.
 16. Thecomputer-implemented method of claim 12, wherein the untagged element islacking the adequate descriptive attribute when said element isassociated with an erroneous tag.
 17. The computer-implemented method ofclaim 12, wherein the ARIA label comprises a text description of theuntagged image.
 18. The computer-implemented method of claim 12, whereinthe code associated with the web page is the DOM or HTML code.
 19. Thecomputer-implemented method of claim 12, wherein the computer systemcomprises one or more computing systems.
 20. The computer-implementedmethod of claim 12, wherein the ARIA label is generated in part by usingan artificial intelligence algorithm operating on a remote server. 21.The computer-implemented method of claim 12, wherein thecomputer-implemented method is performed on a periodic basis.